Sparsity in the Context of Support Vector Machines

This paper surveys the significance of sparsity for the Support Vector Machine (SVM) method. The SVM method is a machine learning technique with a wide range of applications, e.g. medical diagnosis, pattern recognition, and clustering. The method is fairly recent; Vapnik is credited with originating it in 1979. We present a general introduction to SVMs in the context of data classification and discuss how the method has evolved to its present state. Throughout this paper, we highlight the impact of sparsity on SVM. Sparsity is an intrinsic part of SVM. The Karush-Kuhn-Tucker (KKT) optimality conditions for the SVM optimization program impose sparsity on the Lagrange multipliers. Finding a small hypothesis space, i.e. one that is sparse in dimension, for a given data set is crucial to the quality of SVM classification. In addition, we discuss the following well known modifications to the SVM method which utilize sparsity: the Reduced Set Method (RSM), Projected Conjugate Gradient Chunking (PCGC), Sequential Minimal Optimization (SMO), and Reduced SVM (RSVM). Finally, we mention an equivalence between SVM and Sparse Approximation, which may be a first step toward SVM performance guarantees.